Author Topic: The future of GMG....  (Read 18871 times)

0 Members and 1 Guest are viewing this topic.

Offline Dungeon Master

  • Benevolent Dictator
  • Administrator
  • *
  • *
  • Posts: 370
  • Head Honcho
  • Location: Canberra
The future of GMG....
« on: February 26, 2013, 03:23:58 AM »
Sorry to sound dramatic.

GMG has been chewing up hard drive space on the server, at an increasing rate. I am doing some investigating to determine if this is some sort of bug or has the server been hacked and being used by others, or is it just the result of normal use. It has consumed 5GB in the last 2 weeks! We reached our account limit back then (Feb 17) leading to a server crash.

I paid for more HD space, but we reached our (new) limit again today, resulting in more server downtime.

I have now paid an extra $1100 for a much more advanced server package - more processors, more RAM and, crucially, more HD space. Note that this $1100 is on top of my previous package cost. I have re-booted the server under the new package, and all seems good for now.

The bottom line is this: I'm seriously not sure if I can continue to afford to run GMG. I have pre-paid for this calendar year, so rest-assured GMG is safe until Dec 31, 2013. After that, I'm not sure.

The Plan:
Most importantly, I need to investigate if the sudden increase in HD usage is legitimate or otherwise.  I find it hard to believe that GMG existed on 40GB of HD space for many years, and just this past few months HD space is being eaten up at a much, much faster rate.

If I find a bug or simple cause, I will rectify, or ask my hosts to do so. However, if the extra server resource usage is simply from more activity on the forum, then I will have to reconsider the future.

Please note that this is not a sob-story to get more subscribers or other financial aid - please do not send me money based on this post alone. It is possible that there is a simple explanation, and I will find the excess data (maybe a bloated log file, perhaps old backups not being deleted), delete it and we can go back to a cheaper hosting plan. This post is simply to keep members informed.

Its late at night here in Australia, I've stayed up the get the server back on its feet. I will sleep on it and post more in the morrow.

cheers, and happy listening
Rob
 Macs are better. They just are.

Offline Est.1965

  • Veteran member
  • *
  • Posts: 3898
  • Havergal Brian wrote two symphonies in 1965
  • Location: Clydebank, Scotland
Re: The future of GMG....
« Reply #1 on: February 26, 2013, 03:47:52 AM »
 :o
Thanks Rob. 
I am in no doubt that it can be saved one way or another - with the details you've given us it sure seems like some kind of fault in things - but if it IS a genuine problem with so many posts, etc, and GMG being so popular (!), it may be that subscriptions will up, etc, (I'll go Gold shortly), and if not, maybe someone with infinite resource can buy the forum and keep it going.  Or something.   :-[
Most appreciated that you are keeping us so well informed.
John
Dear Hans Rott
In the 1980s there was a creative punk group called "Big Audio Dynamite".  I have decided to apply the term to you, my man.  And I still haven't properly finished your Screenplay yet.  Too bad.  Take care anyway old chum, I'm off to listen to Brahms!
Kind regards, John

Offline springrite

  • Veteran member
  • *
  • Posts: 6273
  • Location: Flying all over the place
  • Currently Listening to:
    Lots of Bach, Brian, Mahler, Rubbra, Beethoven and Buddhist chants
Re: The future of GMG....
« Reply #2 on: February 26, 2013, 03:53:56 AM »
I think John (Mirror Image)'s abstinence from CD purchase alone will save enough money to run GMG.

All kiddings aside, once we know what we are dealing with, I am sure we can come up with a solution.
Do what I must do, and let what must happen happen.

Offline Brian

  • Veteran member
  • *
  • *
  • Posts: 20900
    • Brian's blog
Re: The future of GMG....
« Reply #3 on: February 26, 2013, 05:37:50 AM »
This board replaced the old board in April 2007, so if we've relied on 40GB for those 5 years 11 months (have we? how many times have we upgraded?), that's an average of 563 MB per month, or about 130 MB per week. If we've used 5 GB in the past 2 weeks, that means each week we used up nineteen times our running weekly average.

Now unless somebody's been attaching huge images to every post they see, I have a hard time believing that ordinary forum growth, such as we have already experienced during e.g. the Havergal Brian Proms mania, can account for nineteen times the data use. So I hope that this is indeed some sort of bug or technical glitch. I can't say I "hope" it's hacking, but it sure seems possible.

Offline Papy Oli

  • Veteran member
  • *
  • Posts: 3643
Re: The future of GMG....
« Reply #4 on: February 26, 2013, 06:12:41 AM »
Bug-finding notwithstanding, why wouldn't we actually cull the threads on some of the older pages in each section to free some space as well:

E.g.
Name that tune is 10 pages - cull the last 5
Introductions is 18 pages - cull the last 9
Diner is 173 pages - cull the last 50 or something like that
etc...

Actually delete the Old Listening/Purchasing thread in 2/3 months time as well ?

Bit drastic for sure but how many times would we actually go back to those ? That could maybe help and keep the General / Recordings / Composers as they are, which is the bread and butter of GMG...

PS : I didn't disregard Composing and Opera/Vocal on purpose... :-[ you see what I mean I hope...  :P
« Last Edit: February 26, 2013, 06:19:29 AM by Papy Oli »
Olivier

DavidW

  • Guest
Re: The future of GMG....
« Reply #5 on: February 26, 2013, 06:25:36 AM »
I agree with Papy.  The old posts can be archived as txt only per thread, put on a zip file and shared on mediafire or some such place and then deleted off of the forum. 

This issue could be images or video.  It wouldn't be a bad idea to temporarily block uploading of images and see if that alleviates the current high usage.

Another thing that can be done that is less aggressive than deleting old posts, is to simply delete old images.  Text is insignificant in size next to images.

Offline Opus106

  • Veteran member
  • *
  • Posts: 9043
  • Bachafugaholic
  • Location: Chennai, India
Re: The future of GMG....
« Reply #6 on: February 26, 2013, 06:29:24 AM »
Another thing that can be done that is less aggressive than deleting old posts, is to simply delete old images.  Text is insignificant in size next to images.

There are very few who upload images/files to GMG, as opposed to those who hotlink them or use a file-sharing service.
Regards,
Navneeth

DavidW

  • Guest
Re: The future of GMG....
« Reply #7 on: February 26, 2013, 06:32:05 AM »
There are very few who upload images/files to GMG, as opposed to those who hotlink them or use a file-sharing service.

What should the conclusion be?  That the equivalent of the entire library of congress is being posted every week?  Images and other files are much more likely than text alone.

Offline The new erato

  • Veteran member
  • *
  • Posts: 14561
Re: The future of GMG....
« Reply #8 on: February 26, 2013, 06:34:10 AM »
Not to deny that this problem is real, but I fail to understand why in this day and age the price of storage should be a deterrent to a board like this....?

Offline Opus106

  • Veteran member
  • *
  • Posts: 9043
  • Bachafugaholic
  • Location: Chennai, India
Re: The future of GMG....
« Reply #9 on: February 26, 2013, 06:37:12 AM »
What should the conclusion be?  That the equivalent of the entire library of congress is being posted every week?  Images and other files are much more likely than text alone.

Does a hotlinked image take up as much GMG-space as an attachment? I don't think so. Although if it does, I'm all for making WAYLT text-only. ;D
Regards,
Navneeth

Offline k a rl h e nn i ng

  • Veteran member
  • *
  • *
  • Posts: 51974
  • Et quid amabo nisi quod ænigma est?
    • Henningmusick
  • Location: Boston, Mass.
  • Currently Listening to:
    Shostakovich, Frescobaldi, Stravinsky, JS Bach, Liszt, Chopin, Haydn, Henning
Re: The future of GMG....
« Reply #10 on: February 26, 2013, 06:51:58 AM »
Does a hotlinked image take up as much GMG-space as an attachment? I don't think so. Although if it does, I'm all for making WAYLT text-only. ;D

Well, but I am guessing that the asin tags must result in sales at Amazon, and hence in some incremental benefit back here to GMG.

Actually, I needn't guess at that, having made more than one purchase as a result of seeing such an image and mashing that puppy . . . .
Karl Henning, Ph.D.
Composer & Clarinetist
Boston MA
http://www.karlhenning.com/
[Matisse] was interested neither in fending off opposition,
nor in competing for the favor of wayward friends.
His only competition was with himself. — Françoise Gilot

DavidW

  • Guest
Re: The future of GMG....
« Reply #11 on: February 26, 2013, 07:02:45 AM »
Does a hotlinked image take up as much GMG-space as an attachment? I don't think so. Although if it does, I'm all for making WAYLT text-only. ;D

Don't be sarcastic, people are obviously attaching more images than you think they are.  I doubt that it is even a majority of users that use the asin tags rather than posting images.

DavidW

  • Guest
Re: The future of GMG....
« Reply #12 on: February 26, 2013, 07:04:47 AM »
And besides that is the purpose of testing it.  If all images are disabled and the rate doesn't go down, then I'll be wrong.  But if it does, then we've found the culprit and can resolve the situation without canning gmg at the end of the year.  I'm not saying that I'm right, I'm saying that we have to isolate and test the most obvious culprits.

Offline Gurn Blanston

  • Haydn: that genius of vulgar music who induces an inordinate thirst for beer - Mily Balakirev (1860)
  • Global Moderator
  • *
  • Posts: 31494
  • Support your local Haydn Society
    • Gurn's Haydn Blog
  • Location: Texas, where else?
  • Currently Listening to:
    Haydn, I reckon.
Re: The future of GMG....
« Reply #13 on: February 26, 2013, 07:06:51 AM »
Don't be sarcastic, people are obviously attaching more images than you think they are.  I doubt that it is even a majority of users that use the asin tags rather than posting images.

I post 100% of my pictures either as links to Photobucket or as ASIN links. However, I see dozens of pictures every day that appear to be being hosted here as uploads. Clearly that is not the preferred method, even though it works. Also, if you DO upload to here, take a minute in your photo software and reduce picture size a little bit. Some of those boys are HUGE!  :)

8)
Help support GMG by purchasing from Amazon using this link

Visit my Haydn blog: HaydnSeek

Follow me on Twitter @GurnBlanston106

Offline Opus106

  • Veteran member
  • *
  • Posts: 9043
  • Bachafugaholic
  • Location: Chennai, India
Re: The future of GMG....
« Reply #14 on: February 26, 2013, 07:11:55 AM »
Don't be sarcastic,

Sorry, I wasn't even trying to be.

Quote
...people are obviously attaching more images than you think they are.

Wow, I must've been away from GMG for a long time.


Now, that's sarcasm. ;D ;)


Seriously: unless I'm mistaken, attachments appear beneath the poster's signature (or thereabouts) and never in the main part of the post. Now, how often have we come across attachments? And when we do, how often is it it's more than a few hundred KB, at most? Also, attachment is available for subscribers only, not everyone.
« Last Edit: February 26, 2013, 07:17:19 AM by Opus106 »
Regards,
Navneeth

Offline Sergeant Rock

  • Veteran member
  • *
  • *
  • Posts: 22111
  • Location: Wine Country Germany
Re: The future of GMG....
« Reply #15 on: February 26, 2013, 07:27:16 AM »
In fact, almost no one attaches images. To confirm that I went back through several of the image heavy threads. The only member who routinely attaches images is listener. One man does not a crisis make. Everyone else simply links to images.

I post 100% of my pictures either as links to Photobucket or as ASIN links. However, I see dozens of pictures every day that appear to be being hosted here as uploads.

Where do you see those dozens of huge pictures, Gurn? I must be blind  ;D

Sarge
the phone rings and somebody says,
"hey, they made a movie about
Mahler, you ought to go see it.
he was as f*cked-up as you are."
                               --Charles Bukowski, "Mahler"

Offline Sergeant Rock

  • Veteran member
  • *
  • *
  • Posts: 22111
  • Location: Wine Country Germany
Re: The future of GMG....
« Reply #16 on: February 26, 2013, 07:30:09 AM »
Don't be sarcastic, people are obviously attaching more images than you think they are.  I doubt that it is even a majority of users that use the asin tags rather than posting images.

Where are you seeing these posted images, David? Except for listener I can't find them. And I'm not being sarcastic  ;)

Sarge
the phone rings and somebody says,
"hey, they made a movie about
Mahler, you ought to go see it.
he was as f*cked-up as you are."
                               --Charles Bukowski, "Mahler"

Offline k a rl h e nn i ng

  • Veteran member
  • *
  • *
  • Posts: 51974
  • Et quid amabo nisi quod ænigma est?
    • Henningmusick
  • Location: Boston, Mass.
  • Currently Listening to:
    Shostakovich, Frescobaldi, Stravinsky, JS Bach, Liszt, Chopin, Haydn, Henning
Re: The future of GMG....
« Reply #17 on: February 26, 2013, 07:31:19 AM »
I probably attach more images than I ought . . . though (just to explain) I've generally done that where a recording is not asinnable . . . .
Karl Henning, Ph.D.
Composer & Clarinetist
Boston MA
http://www.karlhenning.com/
[Matisse] was interested neither in fending off opposition,
nor in competing for the favor of wayward friends.
His only competition was with himself. — Françoise Gilot

kishnevi

  • Guest
Re: The future of GMG....
« Reply #18 on: February 26, 2013, 07:54:45 AM »
I agree that images are not the problem.  As noted, only one member routinely uploads images, with the result that there now appear on the first (most recent) page of the new Purchases thread,  less than 140k of uploaded images.  And that's a high amount compared to his usual practice.

Some large images may appear to be uploaded but are actually hotlinked--Harry and MI use this method (as can be confirmed by hovering the cursor over one of their images and right clicking "inspect properties".

I too suspect malware or a server bug (is it possible that the server has developed a glitch that causes it to report incorrect data size?)

Offline Mirror Image

  • Veteran member
  • *
  • Posts: 46446
  • Claude Debussy (1862 - 1918)
  • Location: Northeast GA, US
  • Currently Listening to:
    ...Mist floating above the water...
Re: The future of GMG....
« Reply #19 on: February 26, 2013, 08:21:34 AM »
Sorry to hear about this problem, Rob. I hope you'll be able to find the problem and fix it.
“Works of art make rules; rules do not make works of art.” - Claude Debussy