I can’t find a satisfying description of what a stream is in the context of jq. I know some filters operate on a stream, others on an array. It’s not a data type. I can’t assign a stream to a variable. There’s no .json field of “stream” type.
I tried looking in the jq manual on the website, and in the “Syllabus” on Exercism website.
True, the first concept doc should spend more time defining it.
Imagine an actual stream in the forest. Except instead of water flowing, picture numbers flowing. Or any other kind of data. One element after another with no containment. You can stick a pipe in the stream to redirect it into some filter.
That was just off the top of my head. Was it helpful?
This is how I understand it intuitively, but intuition is sometimes wrong.
The only language I’m really good at is Python. This is my point of reference. The questions that come to my head are:
jq streams like Python iterators?,) joins streams, can I use it to “zip” (interweave) two streams and iterate over them? (No, I can’t, this needs something like transpose, to_entries|group_by, or recursion). The key insight is that comma joins streams at their ends.jq pipes and shell pipes are the most similar, I can’t really talk about Python (@IsaacG ?)
jq does have a “streaming” feature to process JSON files, but “streaming” and “streams” aren’t really the same concept. Yes, this is confusing.
The jq tutorial has this to say:
Data in jq is represented as streams of JSON values - every jq expression runs for each value in its input stream, and can produce any number of values to its output stream.
Streams are serialised by just separating JSON values with whitespace. This is a
cat-friendly format - you can just join two JSON streams together and get a valid JSON stream.
are
jqstreams like Python iterators?
Yes, you can think of them like that. Though typically more like generators. Or, much more accurately, like a file handle/file descriptor/network socket. See also Python’s built in IO package, described as " Core tools for working with streams".
do they have anything to do with processing a large file without fully loading it to memory?
Yes. You can stream from a file handle (read a little bit at a time) as you work on the data. Or like a network socket where the data may only be available a little at a time!
are they related in any way related to “streaming” data from network in real time?
Yup. These are all the same “streams”.
if comma (
,) joins streams, can I use it to “zip” (interweave) two streams and iterate over them? (No, I can’t, this needs something liketranspose,to_entries|group_by, or recursion). The key insight is that comma joins streams at their ends.
No idea. Maybe? I’m not sure if there’s any way to open a second stream in jq. There’s usually one input file/stream that jq processes at a time.
I learned today (official manual) that an innocent comma is actually a generator in jq. Spooky.
A stream is also a way to map a list (map is implemented with a stream).