> As far as I understood DSD is pretty much what comes out of a sigma 
> delta converter (ADC) 
 
Yes, but... 
 
> So unlike a high-speed-sampled 1 bit signal which indicates the 
> current level (0 or 1) at the sampling instance the DSD signal are 
> more like an "upupupdownupdowndownup..." sort of signal which tells 
> whether the level at this sampling instance is higher/lower than was 
> already accumulated. 
 
True. But think again about 1 bit sampling. If you do it directly, it'll 
be horribly nonlinear. If you add enough dither, it'll become linear in 
average and work the way you describe, but then the utility signal will 
be completely swamped in noise. The big insight of noise 
shaping/sigma-delta is that it is possible to shape the noise floor so 
that some minute, low-end fraction of the total bandwidth of the 1-bit 
signal forms a linear, low noise channel, and the quantization noise is 
moved upwards in frequency, while keeping the signal 1-bit. Nothing else 
changes, so the utility signal is *still* some time average of the 
bitstream. It's just that you don't need as steep a lowpass filter for a 
given level of accuracy in reconstruction because the noise is already 
concentrated in the high end. 
 
Intuitively it's a bit difficult to see why the sigma-delta modulator 
works this way. It's usually analysed statistically and in the frequency 
domain, by inserting a dither source inside the quantization loop, then 
treating the step quantizer as an independent source of noise, and 
finally showing both that the input-output transfer function tends 
towards unity in the utility band and that the closed loop response from 
the dither input to the output tends to a highpass characteristic 
related to the modulator order determining lowpass filter (in your 
words, the "accumulator" embedded in the feedback loop. 
 
I'll try to outline one way of getting it in the time domain. If you 
look at the modulator, the loop tries to keep the output of the embedded 
lowpass/accumulator as close to the input as possible. It always outputs 
a correcting bit and assumes that the output will follow in the same 
direction, reducing the error. This is a form of negative feedback 
similar to that used in analog amplifiers; wrapped around a 
nonlinearity, it counteracts it. From another perspective, the 
accumulator is the ideal synthesis filter and the loop around it is a 
naive optimization process which attempts to synthesize a binary string 
which would drive it along the input waveform. The arrangement is stable 
as long as the monotonicity assumption holds, and this happens in the 
low frequency utility band, so there the process tends towards 
transparency with rising sampling frequencies. Above the utility band no 
energy was present before and now is, so the entire process tends 
towards simple addition of high frequency noise which is designed to be 
filtered out by the reconstruction filter. 
 
Now it is relatively easy to see why the loop behaves as it does, 
because the delta-sigma modulator can be broken in independent halves. 
The lowpass filter inside the loop is basically the optimal, linear 
reconstruction filter we're optimizing the bitstream for, and the loop 
is the optimizer. Reconstruction is done by lowpass filtering because 
that's what the optimizer fitted the bitstream for; the order of the 
lowpass filter/accumulator only changes the *kind* of averaging that is 
optimal, and not the overarching principle. In simple delta coding the 
accumulator and the reconstruction filter are of first order and the 
signal is multibit, in sigma-delta they have order upto six and the 
signal is 1-bit, but both operators are still just different kinds of 
moving averages, capable of emulating one another to a high degree. 
 
Since LTI processing does not shift energy from one frequency band to 
another, LTI filters can always be applied directly to DSD as though it 
was 1-bit PCM. Any high frequency, shaped noise will stay up there and 
be cut out upon reconstruction. Of course any filters will cause the 
delicate cancellation between the utility signal and the added noise to 
fail, so that the intermediate signals will no longer be 1-bit but will 
require normal PCM bit widths. Any nonlinear processing can also cause 
interference between the utility band and the noise component, and 
corrupt the utility signal. That's why processing PCM is so much easier. 
But it's easy to see that going into PCM requires nothing more than 
filtering out the noise and (possibly) decimating into the utility 
bandwidth. (Sony's solution is 8-bit intermediates without decimation 
and reshaping back to DSD for delivery.) 
 
 
BTW, the last way of looking at delta-sigma modulation is more general 
than it seems and quite powerful. It allows us to formulate exactly the 
conditions the system relies on, delineate its boundaries, and consider 
alternatives. For example, we see that the optimizer is a greedy, 
heuristic one and relies on a highly simplified assumption (a monotonic 
input-output relationship). This is where the instability of high order 
(i.e. more efficient) delta-sigma modulators comes from, so going with a 
high order filter and replacing the optimizer with something a bit more 
sophisticated can in theory boost the efficiency of bitstream coding and 
probably sidestep the Lipschitz-Vanderkooy critique of DSD because a 
dither linearized feedback loop is no longer present. It's also evident 
from the structure that in principle delta-sigma isn't limited to 
lowpass reconstruction -- just replace the embedded filter with 
something else and use an optimizer which can cope. Et cetera. 
 
> So in other words it has a 'memory' in it. From what I understood 
> there is a specific bitstream that indicates 0. 
 
Sort of. As long as we reconstruct by LTI lowpass filtering and the 
transmitted signal is 1-bit, the bitstream will need to alternate 
symmetrically about the mean and be as high in frequency as possible. 
Depending on which metric you use, the best approximation to zero output 
will come from an alternating series of zeroes or ones (least mean 
square error: energy is concentrated as high in frequency as it can go 
where the reconstruction filter has cut off maximally) or from a 
top-heavy noise (supremum norm in the frequency domain: the energy 
distribution will have to inversely follow the reconstruction filter 
response). 
 
> Given that (unlike PCM) DSD doesn't have the concept of zero there has 
> to be a way of telling "ok we're going to start at 0 again". So I 
> guess one would have to detect this bitstream to put your PCM signal 
> to 0 and then create the signal based on the up/downs and then 
> downsample based on that. 
 
Both PCM and DSD have a concept of zero in the same sense: the mean 
between the highest and lowest values directly representable. In PCM 
systems that will mostly lie between the two middle sampling steps, so 
neither PCM nor DSD can realize a zero exactly but only as a time 
average of a quantized signal. In practice both systems have finite 
memory, so decoding can start anywhere and the resulting average will 
converge to the right value. Nowadays high quality PCM will be in-band 
noise shaped before delivery as well, so the reasoning about residual 
noise levels and zero patterns is almost identical for the two systems. 
 
PCM and DSD really aren't that different if you push them hard.