Digital Diver censorship

[Genesis 0]

Well, well....

 

Those of you who are on both forums may have seen the recent back-and-forth by Craig and I on storage media and camera convergence (video + still) that is taking place.

 

My comment to craig that "what do you think a Microdrive IS?", relating to it being a Winchester disk (which he strongly implied it wasn't!) drew a threat from the owner of the board to ban me for being "disrespectful."

 

My response to craig thus wasn't posted there - nor will it be.

 

However, what this little incident has done is lead me to believe that there is no objectivity allowed at DigitalDiver. Thus, that which is expressed there has to be passed through a filter of "don't offend, even if someone is flatly wrong", which means I can't trust anything posted there - nor, IMHO, can you.

 

So, I will be here, and on SB, instead.

 

Just thought that those of you who might be on both fora would like to know why there was no follow-up from me - basically, I was told that if I did, I'd be banned.

 

So much for that.

 

In fact, I was told explicitly to come over here if I was "unhappy" with the Grand PoohBah's decision on this matter.

 

So here I am!

[Guest wetpixel]

Genesis -

 

Welcome! I hope never to have to censor people here. We really haven't had much disrespect over here so far, and most comments that stem from vehement disagreement usually sort themselves out (that is, except for the shark feeding controversy, which is a topic that we aren't likely to be able to agree upon in one small forum. ).

[craig 0]

Genesis,

 

I would encourage you to participate both here and DD as many do. There is overlap but also some key differences between the boards. I'm not privy to any conversation between you and any admin of that board (nor would I want to be) but I'm sorry if you felt inclined to leave. I wouldn't think that was their intent. Hopefully you will continue at both sites and get value from each.

 

As for the conversation, I never felt disrespected by anything you said and apologize if you felt that way. I didn't even think there was a heated discussion. If you wish to continue the discussion here or elsewhere, please do. I intended my comments to be constructive and would be happy to clarify them in public or private.

 

Welcome to wetpixel.

[Cybergoldfish 1]

I'll second that welcome...

As both Eric & Craig have said there will always be differences in opinions or interpretations on any forum, but you can either agree or not. We often take the micky, but rarely personal enough to warrant either warning or excile... We all share one interest and if it's not fun or informative, what's the point.

What you will find here however, is more constructive evaluations, objective critique and far less Bahpoo!

[Genesis 0]

Genesis,

 

I'm not privy to any conversation between you and any admin of that board (nor would I want to be) but I'm sorry if you felt inclined to leave.  I wouldn't think that was their intent.  Hopefully you will continue at both sites and get value from each.

 

As for the conversation, I never felt disrespected by anything you said and apologize if you felt that way.  I didn't even think there was a heated discussion.

I didn't either Craig, but the owner of the board did, and emailed me. Believe me, I've been on Usenet since '83 or so, and if I wanted to be argumentative or disrespectful, I'm quite capable of it!

 

I asked him POINTEDLY what he objected to, and he quoted the one line that I cited here.

 

How that could be considered "disrespectful" is beyond me, but I was told in no uncertain terms that the "tone" of that line was unwelcome and if I did it again, I would be permanently banned.

 

I had a HUGE response penned to your follow-up at that point, and aborted sending it, as I'm sure it too would have been deemed "over the line."

 

On answering back to him, I got in part, the following:

"I find it incredible that I have allowed myself to be drawn into this stupid debate." and...

"If not, go to wetpixel. Craig posts there as well and you can engage in all of the juvenile one upmanship that your heart desires. "

 

Ok. I'm here.

 

Anyway, I'm not going to pee in his cornflakes. There's no point; if open, honest dialogue isn't welcome there, then it isn't. That's a shame though, because it basically says that anything expressed there has to be taken with a grain of salt - unfortunate, as that destroys the very value of a webboard when people come looking for a debate - which, if you're trying to figure something out, you probably will be looking to see and/or participate in!

 

Glad to hear its not an issue here...

 

BTW, the message I was about to post was a detail of my 22 years of buying winchester-technology media, the progression of speed and performance over that time, and the fact that, today, the 60gb disk in my current laptop will handily fit in the space that my DVCam uses for its tape transport - with room left over. And oh, by the way, my current 1GB CF card - a Ridata - can sustain 3.5MB/sec I/O speeds..... I have in fact DV directly to it as a test! If you were to stack 1GB CF cards (minus the connectors, of course, since you wouldn't do it without stripping all the external BS away) in the tape compartment of a DV camcorder, you'd have a LOT of them in there... and there are, today, 2GB CF cards - they're just stupid-expensive.

 

12GB worth of storage is about an hour in DV format (roughly.) That would make my 60gb laptop drive hold FIVE hours of DV, which is quite an advantage over tape, especially when you consider two further factors:

 

1. You can get to any of it instantly.

and

2. Nothing prevents you from using lossless compression (e.g. Gzip) except the CPU power to do it in real time, and I bet DV compresses VERY well; you almost certainly could achieve 2:1 reduction in storage, and perhaps quite a bit more, without any loss in quality at all. The trade-off is that you would need a file structure and the camera would have to split up the stream into "files", otherwise you would lose the "random access" nature of things, since a GZip'ed stream cannot be entered in the middle and decompressed.

 

10 hours in a removable camcorder capsule, reusable indefinitely, for a couple hundred bucks sounds REALLY good and VERY competitive with tape. This is even more so if you can plug it into your NLE platform (e.g. a FireWire interface?)

 

Now 'ya have the gist of my reply that never got posted, but without all the details.

[SharpDiver 0]

In defense of my partner and our forum, here, in its entirety, is the private message sent to Genesis to point out that his defense of 5050 as a video platform was getting a bit contentious.

 

As you may have read, I have been on the road for the past two weeks, and not actively monitoring the forum.

 

We're glad you joined and appreciate and value your participation, but I have noticed a disturbing tone to your posts, particularly your last post in http://www.digitaldiver.info/yabbse/index....g29241#msg29241.

 

We don't have many rules here, but we do expect and require respect for others and their opinions.

 

If you've visited Wetpixel or Rodales d2d you know that things can get very contentious and nasty there. When we started this board, we specifically set out to avoid that atmosphere, and have done so very successfully.

 

Conflicting opinions are fine, and expressing them is fine, but please tone down the rhetoric a notch or two. Nobody is an expert about everything, and nobody knows everything about any given subject, all we have are opinions, so please keep that in mind when posting.

 

Thanks,

Richard                                                         

 

I certainly did not, nor do I think Richard intended for this to spill over to wetpixel, and I apologize for the intrusion.

 

I will not be sharing some of Genesis comments back to Richard and I, but suffice it to say, he had a decidedly different "tone of type" than he has used in opening this can of worms here.

[craig 0]

As I recall, the discussion was whether it was possible to fit an 8-10GB hard drive into the space of a CF card, not whether a laptop drive could be fit into a video camera. In actuality, the original claim was that DV tape lacked the performance required for digital still use. Clearly, 2.5" and 3.5" drives vastly outperform DV tape, but CF devices do not (yet). Perhaps you have a CF card that matches DV tape. I do not, but my original point was that DV tape would have adequate performance for still use and that is true nonetheless.

 

In any event, there has been some talk of embedding a hard drive in place of tape in video cameras, I've posted a link to one such HD device here in the past. Once you get past a certain data rate requirement, hard drives are your choice. I've also heard of ganging multiple CF flash cards for video use but that seems too expensive to me.

 

I think the future of digital video will be in higher resolution and possibly in higher data rates. Some want to see the tape replaced in favor of an optical disk or solid state, but I don't think consumers will pay a big price for that. There's no denying that removing a hard drive from a camera and plugging it in to your PC/mac is appealing, but it won't play well to the guy who doesn't edit. I would personally love it but there are things higher on my priority list.

 

It would be easy to do some tests on lossless compression of DV files. Perhaps you would like to do some. JVC squeezes 720p 30 frame HD into less than the DV data rate but not in the manner you suggest. I believe most manufacturers are looking more into MPEG, etc. to achieve higher performance and lower data rates rather than trying to modify or extend the DV format.

[JohnJohnsonIII 0]

I don't get it. I just read the entire thread, and Genesis got the troublemaker label? Hmm...maybe they mistook him for Carter, who made the nice "you seem to give quite a bit of advise for someone who doesn't even have a video camera yet" comment.

Either way, welcome. Be careful, though...Eric is an Olympus 5050 Heretic now but at least he hasn't banned us from the boards yet :-D

 

Peace,

John

[craig 0]

...Eric is an Olympus 5050 Heretic now but at least he hasn't banned us from the boards yet :-D

 

Peace,

John

That's because he's a lazy heretic! I haven't heard that Eric feels that the 5050 is a bad rig, but it's not an adequate replacement for anyone accustomed to an SLR. I have one but use it strictly as a loner. I am donating it to my LDS so that it gets more use.

 

I suspect Eric only feels comfortable with the heavy iron in his hand and heard rumor that he recently obtained some seriously heavy iron for his 1Ds. Let's see some pictures!

[Guest wetpixel]

re: Oly 5050

 

Hey! I do think that is is capable of fantastic results. I think the problem is that I am willing to absorb the cost and weight of a housed SLR, and so I'm used to the cleaner image and the fast shutter response.

 

Having said that, the Oly can get some shots that a housed SLR cannot. So there ya go.

 

--

 

re: wetpixel's "contentious and nasty" environment

 

The % of these sort of posts is EXTREMELY low. There are only a few such threads on this board, and aside from a personal scrape between me and Jeff a long time ago, and perhaps the shark feeding controversy thread, there has been nothing that has left prolonged bad feelings, to my knowledge. It's unfortunate that digitaldiver is spreading the word -- even privately -- about wetpixel having that sort of environment. I disagree about the overall character of our forums here. We are honest, and aren't afraid to state opinions strongly. that's about it.

[Genesis 0]

Well Eric, I frankly don't care.

 

I'm glad I found 'ya here..... and I'm enjoying it!

 

You know what they say about things finding their own level!

 

(Shark feeding? I didn't see that one.... could have some fun with it I suspect.... the only shark feeding I've done has been unintentional when spearfishing )

[Genesis 0]

Actually, I argued that the DV tape did have not only the capacity but might make plenty of sense for digital still use.... not the other way around.

 

However, on closer examination it looks like in the space you can get tape, you can get a hell of a lot more disk!

 

The funny thing is that the same thing happened with DLT in the data world. DLT, prior to the "super" varient that is way new and with an unproven history, used to hold a LOT more on a cartridge than a disk of approximately equal physical size. Well, not any more! Now that same cartridge holds 70GB (compressed, @ 2:1) but I can buy a disk quite cheaply (for about 4x the cost of the tape!) that holds 160GB, and 320GB disks are available in the same formfactor, albiet they are expensive. Even the "Super DLT" though has been surpassed with the disk drive capacities in the same physical size!

 

It no longer makes economic sense to use tape to back up disk due to transfer rate and storage density limitations. Rather, it now makes sense to use RAID arrays for physical redundancy and to back up to a SECOND array, which then has the "capsules" (drives) pulled and stored off-site in a vault. Its both faster AND cheaper to do this than to buy the tape library and media to do it to cartridges with a proper tape rotation! Optical or opto-magnetic media, which used to be all the rage due to random access capabilities, is now a bad (and expensive) joke.

 

The last database storage array I built for a customer (I still do consulting work in this realm), totalling close to a terabyte of online storage, was a Raid-5 array backed up with a second one in exactly this fashion. The second one was then pulled and placed in a vault offsite, and the carriers replaced with the other copy. This rotation was done every week, so in the event of a fire or other catastrophic event, no more than one week's worth of data would be lost and have to be recreated. That was an acceptable risk for that customer. It was VASTLY cheaper to do this than to do the same job to DLT library when hardware, software and media costs were considered. The full copy required about 9 hours (including the snapshot functionality and roll-forward when the backup was complete), and was run overnight Sunday, assuming the source array was full. At "usual" load, which is about half full, it was a 4-hour job. The beauty of this system is that in the event of a fire the backup media is instantly usable and requires no reloading back to working storage, as it IS working storage, its about 4x faster in performance than the same backup done to SuperDLT, and the database doesn't have to go offline - ever!

 

I pulled that 8gb drive I have. It won't QUITE fit in a CF-II form factor, but its pretty close. Then realize that the drive is from the late 90s, and things become more compelling.

 

BTW, the Hitachi drive you referenced is cited as having MORE speed than any CF card on the market. If true, then it will handily support a DV video rate. I believe the claims, as most of the gains in I/O speed on Winchesters have not come from the increased platter rotational rate, but rather from the increased areal density of the bits on the platter.

 

Nor is this about to end. IBM has had in development for several years now technology that will record the magnetic domains "vertically" rather than "horizontally" on the platter, and they are also experimenting with phase-angle encoding. The gains in areal density that can be had with these techniques are almost beyond comprehension If they get it working in production multi-terabyte capacity 3.5" formfactor drives will be quite realistic, and 100GB+ 1" drives with transfer rates rivalling the best of what we have now in the 3.5" form factor will be not only possible, but commonplace. The idea of a wristwatch or PDA - literally - with more storage in it than you currently can fit in your desktop machine is NOT an impossibility.

 

The Ridata 1GB card that I have, which some say is the "fastest" card out there, can handily handle data rates of over 5MB/sec in read mode, and about 4MB/sec in write. It is BLAZING fast, and will definitely handle a direct capture from a DVCam; I've done it as a test (until it ran out of space!) and had no dropped frames.

 

BTW I took a 2GB AVI file this evening (a DV capture of about 16 minutes) and compressed it with gzip, to see how "routine, ordinary" compression tools do on such data. This is a lossless compression that has no knowledge of the contents of the file, and thus can't "cheat" in any material way. The source material was a band halftime show clip in 16x9 format.

 

I timed the compression on my Unix box, which is a 1.6ghz AMD chip; the job was saturated on the disk channel (on a 160GB WD disk - a cheapie, one of my $139 specials recently) at 13 MB/sec (!) - so much for those "cheapie disks" being slow. This is a FILESYSTEM I/O run; just for comparison Windows can't TOUCH that data rate, not even XP on a 2.4Ghz machine (so much for XP's "prowess" with NTFS, etc?)

 

I was consuming about 80% of the CPU on this job, so I still had some CPU remaining. The job required approximately 5.5 minutes, but led only to a savings of about 20% of the storage - far less than I had expected. Whether this is due to the "AVI" format I do not know, but it is significantly less gain than I had expected.

 

For grins and giggles I then ran a disk performance benchmark test that I have laying around (which I wrote several years ago for use at a client site) that measures performance across the entire disk's surface as a raw I/O data rate. I haven't used this on a disk in years - previously, 5-6MB/sec was a pretty good performance turned in on SCSI drives that were being tested at the time.

 

That SAME disk records a RAW I/O speed of 42(!) MB/sec (that's bytes, not bits!) when timed with "dd" (a direct I/O to and from the platter.)

 

42MB/sec is, quite honestly, mind-blowing. It is indeed, however, what kind of sequential SUSTAINED performance I can get from that disk. That's roughly half of the ATA/100's RAW possible bus data rate - damn impressive. To put this in perspective that would sustain TEN DV streams AT ONCE and not drop a single frame! From a 160GB disk that cost me $139 at Sam's Club - I'm flatly astounded.

[craig 0]

Then we are in agreement that DV tapes would do fine for still devices performance-wise.

 

It's interesting to consider future technologies but they are still in the future. DV tape was introduced in 1995? The new Hitachi microdrive may be fast but it's still unavailable and it will still be power hungry. I enjoy going more than two days between battery changes on my Nexus D100. With a microdrive that is shortened to two dives. With my setup, a battery change is quite an ordeal.

 

The reason DV avi files are somewhat compressible is that there is some gas still in them. Type 2 ones (those would be the older ones) have the audio streams duplicated. When you get down to the video and audio data itself you'll find them to be essentially incompressible using lossless means.

 

Notebook hard drives are typically 2.5". There are some 1.8" like the ones in the iPods but until recently their capacity was quite limited. CF type II cards allow for a single 1" platter. I don't think you'll fit any existing drive in that space.

 

Enterprise storage gets pretty far afield but it is a subject of which I'm familiar. It is true that tapes make poor financial sense and that meny large customers would like to switch to hard drive-based backups, but few have actually made that switch because (1) they haven't figured out how, and (2) they aren't cost sensitive. Individual customers may do that, but it is not currently an industry trend. Remember, these guys are buying EMC.

 

Hell, I stopped using tapes 6 years ago. I use two RAID 0 arrays on separate physical machines and rsync them nightly. Of course, I don't have a large off-site requirement either. The 1TB club is getting less exclusive all the time. I've been a member for a couple years now and have never discarded a video project since 2000. Hard disk space is cheap and tapes are a PITA. Toss in a couple of DVD-R's for critical data and you are set.

[Genesis 0]

Well....

 

One of the reasons I have work pretty much any time I want it (I'm semi-retired) is that I know how to do things that many other folks think are "hard"

 

Lots of folks thought it was "hard" to put together data networks and ultimately, consumer Internet access. I thought it was pretty easy, and made a dandy business out of it.

 

The 1" CF-II form factor is mostly a solution without a problem. The amusing thing (for me anyway) is the number of device makers who have moved AWAY from it in what looks to be a pretty brazen attempt to associate even more of their customer's money with their wallets. xD is one of those..... I love how Olympus, for example, will only do "stitching" of panoramic shots for you in the camera if you use THEIR xD cards!

 

(That little 8gb disk I was referring to is in a "subnotebook"; its a very, very small disk and was made pretty much only for this line of machine, as was the RAM for it. That's a big part of why that computer is now my deco planner on my boat rather than my "prime" notebook - it simply can't keep up with today's demands as a 'real" workstation. )

 

Now in the "subcompact" camera category I do understand - you need small storage! But in the extreme case you're better off just building the flash into the camera and forgetting about it - the connectors take enough space that it becomes a factor. Even CF, which is relatively-speaking large, the connectors consume a significant amount of the total space, and so does the packaging.

 

But consider the near-miraculous "mini" DV-style camcorders such as the PC-330 and TRV-33. They are marvels of electronics, really. The view LCD screen isn't any great shakes, but getting the electronics required, along with the optical path, in the size of the case is really something, when one considers the fixed overhead of the tape transport itself.

 

But is the transport a "fixed" requirement? Not really.

 

Power consumption is indeed an issue, and there you do have a point - to a point anyway. But that problem exists mostly because we are adapting "common" devices to "uncommon" uses. Power in a disk drive is mostly consumed by the spindle motor, and secondarily, by the head voice-coil drive motor. Making either faster costs lots of power. But do you NEED faster for the head positioner? Not if you're recording DV or stills and do your own filesystem management, so that "long excursions" simply don't happen in normal use. Do you need fast rotation? Only if the areal density is relatively low. As density goes up, performance achieveable at a given rotational rate also rises, and the requirement to spin the platter quickly goes down. A good part of the losses in the motor are due to bearing friction, which Hitachi (and others) have gone a long way towards reducing with the newer fluid bearings - their impetus to develop these has been heat rejection concerns as platter RPM rises, but exploiting that is quite possible even at lower RPMs, where the savings in power and heat are even more pronounced.

 

Flash seems ungodly expensive, but is it really? Look in perspective here - a few years ago 256GB flash cards were prohibitively expensive. My Ridata 1GB card cost me ~$200. I paid roughly that much for a 40MB card a few years back!

 

There is nothing preventing Sony or others, over time, from taking a "pack" of CF chips and making a solid-state "disk" that would replace the tape transport in a DVCam. It would be a VERY cool development if the cost was right. Today, it would be a several-thousand-dollar option, and nobody would want it. But in a couple of years that same configuration might run $200. Now, it suddenly makes sense - for the cost of 20 or 30 tapes in a retail store, you can have five or ten tapes worth of capacity IN THE CAMERA, and swap for ANOTHER ten any time you'd like! It would look like, to your camera or computer, just like a huge disk drive.

 

Consider the advantages - immune to vibration, dust, and other perils of tape. No "dew" considerations. It never (well, ok, close to never - 100,000+ write cycles) wears out, there are no dropouts, and no need for head cleaning (or head replacement), irrespective of the amount of use. Storage is in the 5-10 HOUR range, making the one-hour miniDV limit (which really IS kind of a PITA in many applications) disappear. Power consumption reductions (almost ZERO when reading and literally zero when quiescent) makes the camera batteries last two or more times as long as we get now. Random access, of course. No need to "capture" from your DVCam to your NLE editing software; your NLE software can read the pack DIRECTLY, or you can just copy the pack to your fixed storage on your computer, saving immense amounts of time when putting raw footage into your NLE software for editing and production purposes. It now becomes possible to partially environmentally seal the camera, much as some of the digital SLR makers have done in an attempt to reduce moisture concerns, which is very cool around the underwater environment. Make the interface to it USB2 (its more than fast enough), have it show up as a storage-class device, and you have LITERAL "plug and play" connectivity with the camera AND your computer, requiring only a cable on the computer side.

 

"Internal" conversion/burning support now also becomes trivially easy. Put a second USB interface on the camera (with a custom connector for size reasons, of course) and make a simple DVD burner interface for DIRECT dumps from camera storage to DVDs! Simple "mark in/mark out" editing on the way to the disk becomes trivially easy to do, since you have random access. USB burners are available now, are portable, and the connector is a "nice" size. That would DEFINITELY sell - look at the people who currently want to buy the DIRECT DVD recorder cameras, despite the fact that their video performance is comparatively ratty by DV standards, as they record in MPEG all the time.

 

I was in Circuit City this weekend and watched about a half-dozen customers playing with camcorders (I was oogling the new Sony WEGA LCD HD televisions ) Out of those, essentially all were gravitating towards the two camcorders that recorded directly onto the little DVD recordable disks. Why? Convenience. It won over performance. Yeeeccchhhh says I, but Granny wants convenience for shooting movies of her kids school play!

[Tio Loco 0]

re: wetpixel's "contentious and nasty" environment

 

The % of these sort of posts is EXTREMELY low. There are only a few such threads on this board, and aside from a personal scrape between me and Jeff a long time ago, and perhaps the shark feeding controversy thread, there has been nothing that has left prolonged bad feelings, to my knowledge.  It's unfortunate that digitaldiver is spreading the word -- even privately -- about wetpixel having that sort of environment.  I disagree about the overall character of our forums here.  We are honest, and aren't afraid to state opinions strongly.  that's about it.

Eric, I would like to apologize for that characterization, even though it was made in private. My basis for that impression is a holdover from the days before Digital Diver, when less than cordial exchanges were quite common on Wetpixel.

 

I have not spent enough time here in the past year to know one way or another, so please accept my sincere apology, and my assurance that it will not happen again in public or private communications.

[markprior 0]

Following on from your discussion about hard drive storage and camcorders -

 

Panasonic have recently released the SV AV 100 which is a tiny camcorder which uses a SD card to record MPEG-2 or 4 video. I believe MPEG-2 storage on the 512mb card provided is only around 20 minutes.

 

However the SD card roadmap goes up to 4gb or higher I believe which would certainly give adequate storage.

 

The only issue here isin terms of archiving footage - you would need a fairly large hard drive in your laptop if away on holiday.

 

A review of this can be found at Review of SVAV100