Magento Payment Redirect Order

How about extending the _validate() method on Mage_Sales_Model_Service_Quote, and throw an error there so that it never gets to the "$transaction->save();" bit.

public function submitOrder()
    {
        $this->_deleteNominalItems();
        // do some check here
        $this->_validate();
        // End checks
        $quote = $this->_quote;
        $isVirtual = $quote->isVirtual();

        ........

        try {
            $transaction->save();
            $this->_inactivateQuote();
            Mage::dispatchEvent('sales_model_service_quote_submit_success', array('order'=>$order, 'quote'=>$quote));
        } catch (Exception $e) {
            ...........
        }
        ...........
        return $order;
    }

Validate function looks like this:

protected function _validate()
{
    $helper = Mage::helper('sales');
    if (!$this->getQuote()->isVirtual()) {
        $address = $this->getQuote()->getShippingAddress();
        $addressValidation = $address->validate();
        if ($addressValidation !== true) {
            Mage::throwException(
                $helper->__('Please check shipping address information. %s', implode(' ', $addressValidation))
            );
        }
        $method= $address->getShippingMethod();
        $rate  = $address->getShippingRateByCode($method);
        if (!$this->getQuote()->isVirtual() && (!$method || !$rate)) {
            Mage::throwException($helper->__('Please specify a shipping method.'));
        }
    }

    $addressValidation = $this->getQuote()->getBillingAddress()->validate();
    if ($addressValidation !== true) {
        Mage::throwException(
            $helper->__('Please check billing address information. %s', implode(' ', $addressValidation))
        );
    }

    if (!($this->getQuote()->getPayment()->getMethod())) {
        Mage::throwException($helper->__('Please select a valid payment method.'));
    }

    return $this;
}

The extended function can look like this:

    public function __construct(Mage_Sales_Model_Quote $quote)
    {
        $this->_quote = $quote;
        parent::__construct($quote);
    }

protected function _validate()
    {
        // Code to test comes here
        Mage::throwException(Mage::helper('payment')->__('unsuccessfull.....'));
        // Code ends, now call parent

        return parent::_validate();        

    }

This is quite a common issue during Payment Module development.
Magento offers two hooks for payment method classes to provide redirect URL's, one before the order is created, one after.

If a payment method model implements getOrderPlaceRedirectUrl() the customer will be redirected after the confirmation step of the one page checkout, the order entity will be created.

If a payment method model implements the getCheckoutRedirectUrl() method, the customer will be redirected after the payment step of the one page checkout, and no order entity is created.

This is not ideal, but thats what Magento offers out of the box.

As I said - giving sample that I used for this solution at final.

I prefered to observe event to do post request. Really if you'll use method presented here you will take the same effect, but I prefer to use event observer. So:

First add some data to config.xml to create event observer in frontend section

<events>
    <sales_model_service_quote_submit_before>
        <observers>
            <lacpaycs>
                <type>singleton</type>
                <class>OS_LacPayCS_Model_Observer</class>
                <method>lacpaycs_payment_send</method>
            </lacpaycs>
        </observers>
    </sales_model_service_quote_submit_before>
</events>

then we must create Observer class in OS/LacPayCS/Mode/Observer.php:

class OS_LacPayCS_Model_Observer {

protected $_code = 'lacpaycs';

// Here some our additional functions

/**
 * @param Varien_Object $observer
 */
public function lacpaycs_payment_send(Varien_Object $observer)
{
    /**
     * @var Mage_Sales_Model_Order $order
     * @var Mage_Sales_Model_Quote $quote
     */
    $order = $observer->getOrder();
    $quote = $observer->getQuote();
    $payment = $order->getPayment();
    if ($payment->getMethodInstance()->getCode() != $this->_code) {
        return;
    }

    $helper = Mage::helper('lacpaycs');
    try {

        // Here we prepare data and sending request to gateway, and getting response

        if (!$this->_validateResponse($response)) {
            Mage::throwException('Error '.$this->errorMsg);
        }
    } catch (Exception $e) {
        Mage::throwException($e->getMessage());
    }
}

}

So in two words what we doing here $_code is the same that in our payment model and with it we checking in observer if we catched event when customer using our payment method

All another code is simple, so I think it's no need to comment it